Towards Semantics-Enhanced Pre-Training: Can Lexicon Definitions Help Learning Sentence Meanings?

Self-supervised pre-training techniques, albeit relying on large amounts of text, have enabled rapid growth in learning language representations for natural understanding. However, as radically empirical models sentences, they are subject to the input data distribution, inevitably incorporating bias and reporting bias, which may lead inaccurate understanding sentences. To address this problem, we propose adopt a human learner's approach: when cannot make sense word sentence, often consult dictionary specific meanings; but can same work models? In work, try inform pre-trained masked meanings semantics-enhanced pre-training. achieve contrastive holistic view meanings, definition pair two related words is presented model such that better associate with its crucial semantic features. Both intrinsic extrinsic evaluations validate proposed approach semantics-orientated tasks, an almost negligible increase training data.